[KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown by dajiaohuang · Pull Request #7797 · apache/kyuubi

dajiaohuang · 2026-10-26T19:12:54Z

Why are the changes needed?

When SparkContext stops before engine shutdown reaches stopServer(), the latch in SparkSQLEngine.main may never be released. The main thread polls the latch and checks SparkContext; if it is stopped first, the code now throws KyuubiException through the existing engine error path. That path records the abnormal termination and diagnostics instead of returning normally (which could make YARN report the application as succeeded). Earlier PRs #7592 and #7680 are closed and unmerged.

How was this patch tested?

git diff --check passed for this review follow-up.
Maven/Spotless and the Spark test suite were not rerun for this revision because the protected shared Zinc-cache incident prohibits Maven reruns. The earlier 3/3 SparkSQLEngineSuite run was on commit e24adad, before this follow-up, and does not validate the current diff.

Was this patch assisted by generative AI tooling?

Assisted-by: Pi worker dev-w11@stew-c1-4w (openai-codex/gpt-6-luna).

github-actions · 2026-10-26T19:21:56Z

Hello @dajiaohuang,
Thanks for finding the time to submit the Pull request, the Kyuubi community thanks you！
We really appreciate the community's efforts to improve Apache Kyuubi.
If you are new to Apache Kyuubi for code changes, please take some time to read our contributing guide.

pan3793 · 2026-10-27T06:12:37Z

can you change it to the #7680 version? I feel that is much readable?

dajiaohuang · 2026-10-27T21:26:34Z

Thanks for the suggestion. I simplified the wait helper to use the one-second timed-latch loop from #7680, with the stopped-context check and warning.

I intentionally exit the wait rather than call currentEngine.stop() again on every timeout. SparkSQLEngine.stop() is guarded by shutdown.compareAndSet(false, true), so if shutdown is already stalled, another call is a no-op and leaves the main thread waiting. Returning lets main continue through its finally cleanup.

The targeted SparkSQLEngineSuite passed 3/3 for the behavior change. I added only a comment afterward; no Maven rerun was done.

pan3793 · 2026-10-28T01:34:20Z

  private val namespace = "n" * 63
  private val podUid = "u" * 36

+  test("[KYUUBI #7590] stop waiting when SparkContext has stopped") {


it's overkill to add such an ut, remove it and inline the def waitForEngineTermination

Done in commit 9ace3f8. I removed the unit test and helper, and inlined the timed wait in main; when SparkContext stops, the method returns through the existing finally cleanup. git diff --check passed. Maven tests were not rerun because of the shared Zinc-cache restriction.

pan3793

Thanks for the fix. A few comments:

Throw KyuubiException instead of return when SparkContext is stopped, see inline.
Revert the unrelated blank-line removal.
Please update the PR description to match the final change: the "How was this patch tested?" section mentions a regression test that is no longer in the diff, and the "without calling engine.stop() again" wording will be stale after (1).

pan3793 · 2026-10-28T01:59:01Z

+            if (spark.sparkContext.isStopped) {
+              // stop() may already be running, so exit the wait instead of calling it again.
+              warn("SparkContext has been stopped, breaking main thread wait")
+              return


Throw a KyuubiException here instead of returning. If SparkContext stops before the latch is released, the engine is terminating abnormally. A normal return makes YARN report the app as SUCCEEDED and spark-submit exit 0. Throwing also reuses the catch below: it calls engine.stop() (a no-op if a stop is already in flight, because of shutdown.compareAndSet), posts the end EngineEvent with diagnostics, and logs at error level.

while (!countDownLatch.await(1, TimeUnit.SECONDS)) { if (spark.sparkContext.isStopped) { throw new KyuubiException( "SparkContext stopped before SparkSQLEngine, terminating the engine main thread") } }

Please also update the comment above the loop to match.

Implemented in 17c5766: if SparkContext stops before the latch is released, the main thread now throws KyuubiException and uses the existing engine error path to stop/report abnormal termination. I updated the loop comment as well. Tests were not rerun because the protected shared Zinc-cache restriction remains; the PR description now states that limit.

pan3793 · 2026-10-28T01:59:01Z

  private lazy val user = currentUser

  private val countDownLatch = new CountDownLatch(1)
-


Unrelated change, please revert.

Reverted the unrelated blank-line removal in SparkSQLEngine.scala.

Fixes #7590. ### Why are the changes needed? When SparkContext stops before engine shutdown reaches `stopServer()`, the latch in `SparkSQLEngine.main` may never be released. The main thread polls the latch and checks SparkContext; if it is stopped first, the code now throws `KyuubiException` through the existing engine error path. That path records the abnormal termination and diagnostics instead of returning normally (which could make YARN report the application as succeeded). Earlier PRs #7592 and #7680 are closed and unmerged. ### How was this patch tested? - `git diff --check` passed for this review follow-up. - Maven/Spotless and the Spark test suite were not rerun for this revision because the protected shared Zinc-cache incident prohibits Maven reruns. The earlier 3/3 `SparkSQLEngineSuite` run was on commit `e24adad`, before this follow-up, and does not validate the current diff. ### Was this patch assisted by generative AI tooling? Assisted-by: Pi worker dev-w11stew-c1-4w (openai-codex/gpt-6-luna). Closes #7797 from dajiaohuang/kyuubi-7590-spark-context-stop. Closes #7590 17c5766 [dajiaohuang] Report abnormal SparkContext termination 9ace3f8 [dajiaohuang] Inline SparkContext termination wait e24adad [dajiaohuang] Simplify SparkContext termination wait bfb8d04 [dajiaohuang] [KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown Authored-by: Wu Shuwen <mikewushuwen@outlook.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit d953184) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2026-10-28T03:17:41Z

Merge Summary:

merged into master d953184
merged into branch-1.12 d782637
merged into branch-1.11 71c3624
merged into branch-1.10 2c98f13

Posted by merge_kyuubi_pr.py

[KYUUBI apache#7590][SPARK] Stop waiting after SparkContext shutdown

bfb8d04

github-actions Bot added the module:spark label Oct 26, 2026

Simplify SparkContext termination wait

e24adad

pan3793 reviewed Oct 28, 2026

View reviewed changes

Inline SparkContext termination wait

9ace3f8

pan3793 reviewed Oct 28, 2026

View reviewed changes

Report abnormal SparkContext termination

17c5766

pan3793 approved these changes Oct 28, 2026

View reviewed changes

pan3793 assigned dajiaohuang Oct 28, 2026

pan3793 added this to the v1.10.4 milestone Oct 28, 2026

pan3793 closed this in d953184 Oct 28, 2026

Navigation Menu

Repository navigation

[KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown - #7797

[KYUUBI #7590][SPARK] Stop waiting after SparkContext shutdown#7797
dajiaohuang wants to merge 4 commits into
apache:masterfrom
dajiaohuang:kyuubi-7590-spark-context-stop

dajiaohuang commented Oct 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

pan3793 commented Oct 27, 2026

Uh oh!

dajiaohuang commented Oct 27, 2026

Uh oh!

pan3793 Oct 28, 2026

Uh oh!

dajiaohuang Oct 28, 2026

Uh oh!

pan3793 left a comment

Uh oh!

pan3793 Oct 28, 2026

Uh oh!

dajiaohuang Oct 28, 2026

Uh oh!

pan3793 Oct 28, 2026

Uh oh!

dajiaohuang Oct 28, 2026

Uh oh!

pan3793 commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		private lazy val user = currentUser

		private val countDownLatch = new CountDownLatch(1)

Conversation

dajiaohuang commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Was this patch assisted by generative AI tooling?

Uh oh!

github-actions Bot commented Oct 26, 2026

Uh oh!

pan3793 commented Oct 27, 2026

Uh oh!

dajiaohuang commented Oct 27, 2026

Uh oh!

pan3793 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

dajiaohuang Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

dajiaohuang Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

dajiaohuang Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dajiaohuang commented Oct 26, 2026 •

edited

Loading